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Abstract. The idea that one might extend numerical linear algebra, the collection of matrix com- 
putational methods that form the workhorse of scientific and engineering computing, to numeri- 
cal multilinear algebra, an analogous collection of tools involving hypermatrices/tensors, appears 
very promising and has attracted a lot of attention recently. We examine here the computational 
tractability of some core problems in numerical multilinear algebra. We show that tensor analogues 
of several standard problems that are readily computable in the matrix (i.e. 2-tensor) case are NP 
hard. Our list here includes: determining the feasibility of a system of bilinear equations, determin- 
ing an eigenvalue, a singular value, or the spectral norm of a 3-tensor, determining a best rank-1 
approximation to a 3-tensor, determining the rank of a 3-tensor over R or C. Hence making tensor 
computations feasible is likely to be a challenge. 



1. Introduction 

There has been a recent spade of work on 'tensor methods' in computer vision, data analysis, 
machine learning, scientific computing, and other areas (examples cited here include [TJ [U QUI QTJ 
[I2[TiE[I71[JHl[JJH[IS also the bibliography of the recent survey (3TJ 

and the 244 references therein). The idea of using tensors for numerical computing is appealing. 
Nearly all problems in computational science and engineering may eventually be reduced to one 
or more standard problems involving matrices: system of linear equations, least squares problems, 
eigenvalue problems, singular value problems, low-rank approximations, etc. If similar problems 
for tensors of higher order may be solved effectively, then one would have substantially enlarged 
the arsenal of fundamental tools in numerical computations. 

We will show in this paper that tensor analogues of several problems that are readily computable 
in numerical linear algebra are NP hard. More specifically consider the following problems in 
numerical linear algebra. Let F = M. or C. Let A £ Y mxn ^ b £ Y m , and r < min{m, n}. 

Rank and numerical rank: Determine rank(A). 

Linear system of equations: Determine if Ax = b has a solution x 6 F™. 
Spectral norm: Determine the value of ||^4||2,2 := max ||x||2=ill^ x ll2 = °"max(^4)- 
Eigenvalue problem: Determine if A 6 F has a non-zero x 6 F n with Ax = Ax (assume 
m = n). 

Singular value problem: Determine if a £ F has non-zero x 6 F n , y G F m with Ax = ay, 
A T y = ax. 

Low rank approximation: Determine A r £ jp mxn with rank(A r ) < r and \\A — ^4 r ||_p = 

m i n rank(B)<rll^ - B\\ F . 

We will examine generalizations of these problems to tensors of order 3 or higher and show that 
they all fall into one of the following categories: unsolvable (low rank approximation when rank 
r > 1; see Section [8]), NP hard to approximate (spectral norm; see Section [7]), or NP hard (the 
remainder) . 

Johan Hastad has shown that tensor rank is NP hard over Q and NP complete over finite fields 
[24] . We will extend the NP hard list of tensor problems to include several other multilinear 
generalizations of standard problems in numerical linear algebra. These multilinear problems are 
not only natural extensions of their linear cousins but have all surfaced in recent applications 
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[zi ei noi nn nzi na nn eh imi eh esi sni sn hhi m]. Let^e F* xmxn be a 3-tens 0r and 

Ak,Bk, Ck £ F nxn , k = 1, . . . , n, be matrices. Our list includes the following: 
Rank and numerical rank: Determine rank(„4). 

Bilinear system of equations: Determine if x T A^y = ctfc, y T B^z = fik, z T CfcX = 7^, 

k = 1, . . . ,n, has a solution (x, y, z) £ P x F™ x F n . 
Spectral norm of a 3-tensor: Determine the value of sup^y^gl^x, y, z)|/||x||2 ||y||2 INIk- 
Eigenvalue a 3-tensor: Determine if A £ F has a non-zero x E F n with A(x, x, I) = Ax 

(assume I = m = n). 

Singular value of a 3-tensor: Determine a G F has non-zero x 6 F', y £ F m , z E ¥ n with 
A(x, y , I) = crz, „4(x, J, z) = cry, A{I, y, z) = ax. 

Best rank-1 approximation to a 3-tensor: Determine (x, y, z) GF'x F m x F n that min- 
imizes ||^4 — x ® y (g) z||_p. 

We have adopted the shorthands A(x, y, z) = X^fc=i aijkXiVjZj k and .A(x, y, /) = SiJLi a ijkXiVj 
(see Section [2]). As is the case for matrices, the definitions of eigenvalues and singular values 
for 3-tensors come from the stationary values of the cubic and multilinear Rayleigh quotient 
„4(x,x,x)/||x||p and ^l(x,y,z)/||x||p||y||p||z||p where p = 2 or 3 (cf. Sections [6] and [7]) . 

While Hastad's result applies to Q and ¥ q , these choices of fields do not make sense for all 
but one of the above problems (the exception being the bilinear system of equations) as these are 
analytic problems only well-defined over a complete field of characteristic with an absolute value. 
Among such fields, R and C are by far the most common in applications and so we shall restrict 
our discussions to these fields. As for the exceptional case, over Q, the feasibility of a bilinear 
system of equations is undecidable, and over ¥ q , they are NP hard in both the Blum-Shub-Smale 
and Cook-Karp-Levin sense. 

In many cases, certain forms of quadratic feasibility and trilinear optimization stand in the way 
of solving tensor problems. Therefore, to demonstrate NP hardness of many tensor problems, we 
prove that these restricted forms of quadratic feasibility and trilinear optimization are at least as 
hard as determining 3-colorability or max-clique, two well-known NP hard problems |20j. 

Throughout this paper, the notion of NP hardness used is the traditional one of Cook-Karp-Levin 
[IH [291 E3], and we assume that all our inputs are rational numbers. We do not need to involve 
the Blum-Shub-Smale [5] framework. 

Note also that all but two problems on our list are decision problems and so questions of poly- 
nomial time approximability do not arise. The two exceptions are the best rank-1 approximation 
problem and the spectral norm problem, which are optimization problems. We will see that the 
former reduces to the latter (Section [8]) and that the latter is NP hard to approximate (Section [7] 
and [25]). 

2. Tensors as hypermatrices 

Let V\, . . . , Vk be vector spaces of dimensions d\ , . . . , c?& respectively over a field F. An element 
of the tensor product V\ ® ■ ■ ■ ® Vk i s called an order-/c tensor or k-tensor for short. Since we may 
always restrict a A;-tensor problem k > 3 to the case k = 3 (by filling a tensor with zeroes) , we shall 
constrain our analysis in this paper to 3-tensors. 

Up to a choice of bases on Vi, V%, V3, a 3-tensor in V\ ® V2 <8> V3 may be represented byalxmxn 
array of elements of F, 

A - VHjklij tk= i G * 

These are sometimes called hypermatrice^ [21] and come equipped with some algebraic operations 
inherited from the algebraic structure of V\ 18) Vi ® V3: 



The subscripts and superscripts will be dropped when the range of i,j, k is obvious or unimportant. 
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• Outer Product Decomposition: every A = [cii/fc] G ]p' xmxn ma y De decomposed as 

X a X a ® y Q <g> Z a , (lijk = > i XaXiaVjaZkc 

a=l ^ — 



va ■ 



with X a G F, Xq, G F',y a G F',z Q G F n . For x = [x lt . . . , x,] T , y = . . . , y m ] T , z = 
[z u . . . , z n ] T , the quantity x <g> y <g> z = lx iyj z k f^ =1 G ¥ lxmxn . 

Multilinear Matrix Multiplication: every A = [tofc] G F ixmxn may be multiplied on its '3 



mxr 



sides' by matrices X = [x ia ] G ¥ 1x p, Y = [y jp ] G F mx< ?, Z = [z kj ] G F 
(2.2) .4 • (X, Y, Z) = \c aM \ G ¥ pxqxr , c aM = >..,, a ijk x ia y^z kl . 

A different choice of bases on V\ , . . . , V k would lead to a different hypermatrix representation of 
elements in V\ (g) ■ • • <8) V k — where the two hypermatrix representations would differ precisely by a 
multilinear matrix multiplication of the form (12. 2p where X, Y, Z are the respective change of basis 
matrices. For the more pedantic readers, it is understood that what we call a tensor in this article 
really means a hypermatrix. In the context of matrices, x (g) y = xy T and A ■ (X, Y) = Y T AX. 
When p = q = r = lin (|2.2p . i.e. the matrices X, Y, Z are vectors x, y, z, we omit the • and write 

El,m,n 
. . dijkXiyjZ^ 

for the associated trilinear functional. We also note the special case when one or more of the 
matrices X, Y, Z in (I2.2p is the identity matrix. For example, 

El,m r 
. . o-ijkXiVj G F n and A(x, I m , I n ) = ) a ijkXi £ ¥ mxn . 

In particular, the (partial) gradient of the trilinear functional A(x, y, z) may be expressed as 

V x .4(x,y,z) =A(Ii,y,z), V y A(x,y,z) = A(x.,I m ,z), V z .4(x,y,z) =„4(x,y,/ n ). 

A 3-tensor S = [sjjfc] G F nxnxn is called symmetric if Sij k = s^y = sji k = Sj k i = s k ij = s k ji.The 
homogeneous polynomial associated with S and its gradient can again be conveniently expressed 
via ([22]) as 

En 
. s ijk XiXjX k , V<S(x,x,x) = 3S(x,x,/ n ). 
^j?™ — * 

Observe that for a symmetric tensor, 5(I n ,x,x) = 5(x, J n ,x) = 5(x,x, I n ). A special symmetric 
tensor is the delta tensor X = [<%fcjf j k=l G F nxnxn where Sij k = li&i = j = k and otherwise. 
The delta tensor generalizes the identity matrix in some restricted sense (unlike F nxn , the set of 
3-tensors F nxnxn is not a ring and 'identity tensor' would be a misnomer). 

A tensor that can be expressed as an outer product of vectors is called a decomposable tensor 
and rank-1 if it is also nonzero. More generally, the rank of a tensor A = [or/'fcJi'Jfc^Li ^ F' xmxra , 
denoted rank(^4), is defined as the minimum r for which A may be expressed as a sum of r rank-1 
tensors [271I2H]. 

(2.4) rank(^l) := minjr A = ^ _^ X a x a ® y a ® z a \. 

The definition of rank in (|2.4h agrees with matrix rank when applied to an order-2 tensor. 

We will now move on to define analytic notions. So in the following, F will denote either R or 
C. The Frobenius or Hilbert- Schmidt norm of a tensor A = kjfcJ-JJ" x G ¥ lxmxn is defined by 



(2-5) \\A\\ F = [£. 

Another common class of tensor norms generalizes operator norms of matrices: 



, _, i 

^^,m,n ^ 



defines a norm for any 1 < p, q, r < oo. For p = q = r = 2, we call ||«4||2,2 2 the spectral norm of A 
in analogy with its matrix cousin. 

The discussion in this section remains unchanged if R is replaced by C throughout. A minor 
caveat is that the tensor rank as defined in (|2.4p depends on the choice of base fields (see [T6J and 
Section [9] for a discussion). 

3. Quadratic Feasibility and 3-colorability 

This section and the next discuss the problems from complexity theory that form the technical 
heart of our paper. A basic problem in computational complexity is to decide whether a given set 
of quadratic equations has a solution over a field F. It turns out that many questions about tensors 
and bilinear systems can be reduced to this situation with F = R or C; thus, the following decision 
problem will play a large role in our complexity analysis of them (Theorems 15.61 16.31 an d 17. 3p . 

Problem 3.1 (Real Quadratic Feasibility QF K ). Let Gj(x) = x T ^4jX for i = 1, . . . , n be n homo- 
geneous, real quadratic forms; that is, x = (xi, . . . ,x n ) T is a column vector of indeterminates and 
each Ai is a real, symmetric n x n matrix. Determine if the system of equations {Gj(x) = 0}™ =1 
has a nontrivial real solution 0/x£ R n . 

We may ask for the complexity of this problem in two ways. The real complexity of Problem 13. II 
refers to number of field operations in R required to determine feasibility as a function of n. On the 
other hand, assuming that the matrices Ai have rational entries, the bit complexity of Problem 13.11 
refers to the number of bit operations necessary as a function of the number of bits required to 
specify all of the Ai. For instance, the real complexity of multiplying two integers of size N (and 
thus of bit size logiV) is 0(1), whereas the bit complexity is 0(log N log log N) (using the fast 
Fourier transform). Throughout this document, we shall refer to this second version of complexity. 

As the following theorem indicates, it is unlikely that there is a polynomial time algorithm to 
solve Problem 13.11 However, we remark that if we fix the number of equations to be m and allow 
the number of indeterminates n to vary, there is an algorithm which is polynomial in n due to 
Barvinok [2]. For an extension to sets of homogeneous bilinear equations, see Section [H 

Theorem 3.2. 

is NP hard. 



Graph 3-coloring is polynomial reducible to Problem \3.l\ Therefore, Problem \3.1 



Remark 3.3. It is well-known that inhomogeneous quadratic feasibility is NP hard over R and NP 
complete over finite fields [1] . 

We first describe some simple observations giving flexibility in specifying a quadratic feasibility 
problem. The proofs are immediate and therefore omitted. 

Lemma 3.4. Let Gi(z) be as in Problem \3.1l Consider a new system -Hj(x) = x T l?jX of 2n 
equations in 2n indeterminates given by: 



B 



2 1 



A;. 






-Ai 



B 



2i+l 





Ai 



A, 




1- 



, n. 



Then the equations {Hj(x) = 0}|™i have a nontrivial real solution 0^x6 M? n if and only if the 
equations {Gi(z) = 0}f =1 have a nontrivial complex solution 0/z £ C™. 

Lemma 3.5. Let Gj(x) = x T yljX for i = 1, . . . ,m with each Ai being a real, symmetric n x n 
matrix. Consider a new system Hj(x.) = x T i^-x of r >m + l equations in s > n indeterminates: 



B; 







1. 



, m; 








j = m + 1 , . 



1: B r 




/ 



in which I is the (s — n) x (s — n) identity matrix. Then {Hj(x) = 0}^ =1 
solution / x £ I s if and only if {Gj(x) = 0}^ have a nontrivial real solution 0^x£ 
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have a nontrivial real 

n 



To prove that Problem l3.2l is NP hard, we shall reduce graph 3-colorability to quadratic feasibility 
over C. The process of turning colorability problems into polynomial systems appears to originate 
with Bayer's thesis [3] although it has appeared in many other places, including the work of de Loera 
[HI [15] and Lovasz [35]. For a recent discussion of these ideas, see the paper [26] . 

Let G = (V,E) be a simple, undirected graph with vertices V = {1, . . . , n} and m edges E. A 
(vertex) 3-coloring of G is a map v : V — >■ {1,uj,uj 2 }, in which u) = e 27rv/rT / 3 E C is a primitive 
cube root of unity. We say that a 3-coloring v is proper if adjacent vertices receive different colors; 
otherwise v is improper. The graph G is 3-colorable if there exists a proper 3-coloring of G. 



Definition 3.6. The color encoding of G is the set of3n+\E\ polynomials in 2n+l indeterminates: 
(3.1) C G :-- 



XiUi Z , UiZ Xj , XiZ , i — 1, . . . , 71, 

x$ + XiXj + x%, {i,j}eE. 

Lemma 3.7. Cq has a common nonzero complex solution if and only if the graph G is 3-colorable. 

Proof. Suppose that G is 3-colorable and let (x\, . . . ,x n ) £ C be a proper 3-coloring of G. Set 
z = 1 and yi = 1/xi for i = 1, . . . , n; we claim that these numbers are a common zero of Cq. It is 
clear that the first 3n polynomials in (|3.ip evaluate to zero. Next consider any expression of the 
form xf + x-iXj + xf in which {i,j} E E. Since we have a 3-coloring, it follows that ^ Xj) thus, 

ry* ^ ry ^ ( ry ^ ry ^ \ / f ry . ry* . *\ rv* ^ I /y 1 . . I ry* ^ 

— J-j — x^i j } I \ t j ) — ~* I 3 ' i ' 

Conversely, suppose that the set of polynomials Cq has a common nontrivial solution, 

( Xl ,...,x n , yi ,...,y n ,z) e C 2n+1 \{0}. 

If z = 0, then all of the X{ and yi must be zero as well. Thus, we have and since the equations 

are homogenous, we may assume that our solution has z = 1. It follows that xf = 1 for all i so 
that (xi, . . . , x n ) is a 3-coloring of G. We are left with verifying that it is proper. If {i,j} € E and 
Xi = Xj, then = xf + XiXj + xf = 3xf, so that Xi cannot be a root of unity. It follows that Xi ^ xj 
for all {i,j} G -E 1 , and thus G is 3-colorable. □ 



Remark 3.8. The last \E\ polynomials in the definition (I3.ip of Cq may be replaced by the following 
\V\ = n quadratics: pi(x) := Y^{i,j}eE( x i + x i x j + x "j)' « = 1, ■ ■ ■ ,n. 

Proof of Theorem \3. 21 Given a graph G, construct the color encoding Cq. Lemma 13.71 savs that 
the set of homogeneous quadratic polynomials Cq has a nonzero complex solution if and only if G 
is 3-colorable. Adding dummy indeterminates to make the system square by Lemma 13.51 it has a 
nonzero complex solution iff the system given by Lemma 13.41 has a nonzero real solution. □ 

4. Trilinear Optimization and Max-clique 

In this section, we describe the technical setup required to prove our complexity results for tensor 
singular values (decision Theorem l7.2l and approximation Theorem l7.5p . Let G = (V, E) be a simple 
graph on vertices V = {1, . . . , n} with m edges E, and let uj = oj{G) be the clique number of G (that 
is, the number of vertices in a largest clique of G). An important result linking an optimization 
problem to the number uj is the following classical theorem of Motzkin and Straus |36[ [I]. 

Theorem 4.1 (Motzkin-Straus). Let A n := {(x 1 , . . . , x n ) £ R^ : YJLi x i = *} and let G = (V, E) 
be a graph on n vertices with clique number oj{G). Then, 

1 x ^ 

1 —r = 2 ■ max > XiXj. 

0J(G) xeA„^{iJ}6B 1 



Let Aq be the adjacency matrix of the graph G. For each positive integer £, define Qe := Aq+jJ, 
in which J is the all ones matrix. Also, let 

mi : = max x Qfii. = 1 H — . 

xeA„ " £oj 

By the Motzkin-Straus theorem, we have = 1 and also that 

(4.1) m e < 1 if £ > u; m e > 1 if £ < oj. 

For k = 1, . . . , m, let = ^Ei k j k + ^Ej k i k in which {ik,jk} is the kth edge of G. Here, the nx n 
matrix E^ has a 1 in the (i,j')th spot and zeroes elsewhere. For each positive integer £, consider 
the following optimization problem (having rational input): 

M * ■= „gj« {E! =1 ( uTrl/u ) 2 + £™ + E™J uTEkU ?} • 

Lemma 4.2. For any graph G, we have Me = me. 

Proof. By construction, Me = j + 2 • max| u || 2= i Y^u j}eE u \ u % which is easily seen to equal me.. □ 

Using an observation of [25], we further note the following. 
Lemma 4.3. 

Proof. The proof can be found in |25j. The argument uses the KKT conditions to find a maximizer 
with v = ±u for any optimization problem involving sums of squares of bilinear forms in u, v. □ 

Next, let 
(4.2) 

T <? := 1 1 ™ ax „ {/ , 1 ( u ^ !v)wi + > (u E k v)we+k + V, , (u £?jfcv)w m+ ^ + jfc ^ . 

I|u||2 = ||v||2 = ||w|| 2 = l [^«=1 ^fc=l ^fc=l J 

Lemma 4.4. If G is a graph with clique number oj = oj(G), then the optimization Te = Te(G) has 
Te = liff£ = u; Te > I iff £ < u; T t <liff£>u. 

1 /2 

Proof. We shall prove that TJ? = M t 1 , from which the lemma follows (using (j4. 1 j) ) . If we fix real 
numbers a = (oi, . . . , a s ) T , then by the Cauchy-Schwarz inequality, a sum of the form X)i=i a « w « 
with 1 1 w 1 1 2 = 1 achieves a maximum value of ||a||2 (with w, = aj/||a||2 if ||a||2 7^ 0). A straightfor- 

1/2 

ward computation using this observation now shows that Te = M* . □ 

As we shall see in Section Lemma 14.41 is a crucial component of our proof of Theorem 17.21 
There, we would like to show that determining if a tensor has a given nonzero singular value is 
NP hard, and the tensors we consider will be those 3-tensors At = Ae{G) with aijk equal to the 
coefficient of the term UiVjWk in the multilinear form (I4.2p . 

5. Systems of Bilinear Equations 

We now apply the ideas of the previous sections to prove the results discussed in the introduction. 
Multilinear systems of equations have been studied as early as in the early 19th century. The 
following result was known to Cayley [8]. 
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Example 5.1 (2x2x2 hyperdeterminant). For A = {a^kl £ C 
Det 2 , 2 ,2(7l) ^ ' 



2x2x2 



, we define 

2 



det f 


1000 


aoio 


+ 


fflioo 


ClllO 


\ - det ^ 




aoio 




aioo 




) 




fflooi 


«011 




OlOl 






aooi 






aioi 


am_ 





4 det 



aooo aoio 
aooi "on 



det 



oioo ano 
Q.101 tsin 



A result that parallels the matrix case is the following: the system of bilinear equations 

fflooo^oyo + aoio^oyi + aioo^iyo + ano^iyi = 0, aooi^oyo + aonZoyi + aioiTiyo + a m iii/i = 0, 

aooozo-zo + aooizo^i + aioo^i^o + aioiiEizi = 0, aoio^o^o + aon^o^i + ai 1Q xiz + nmnzi = 0, 

aoooyozo + aooij/o-zi + aoioyi^o + aonj/i-zi = 0, a 10 oyozo + aioij/o-zi + a 110 yiz + ami/izi = 0, 

has a non-trivial solution iff Det2,2, 2 = 0. 

1, . . . , n, we would like to 



1, . . . ,71. 



More generally, given 3ra coefficient matrices A k ,B k ,C k £ F nxn , k 
solve the system of bilinear equations 

(5.1) x T A k y = a k , y T B k z = fi k , z T C fc x = j k , k = l,...,n. 

We will restrict our attention to the homogenous case where a k = f3 k = ^ k = for k 
We will show that the associated feasibility problem is NP hard. 

Our definition of a bilinear system in (|5.ip is not arbitrary. If the coefficient matrices are obtained 
from a tensor A = [<%■*;] -j jfc=1 £ F nxnxn as A k = [a ijk }f J=1 , Bj = [a ijk ]™ k=l , Q = [a ijk ]™ k=1 , 
i,j,k = 1, . . . ,ra, then there exists a non-zero solution in the homogeneous case iff the n x n x n 
hyperdeterminant of A is zero [21\ [22] — exactly as in the matrix case. 

In this section, we consider some natural bilinear extensions to the quadratic feasibility problems 
encountered earlier. The main result of this section is Theorem 15.6} which shows that the following 
bilinear feasibility problem (tensor quadratic feasibility over M) is NP hard. In Section [3 we use 
this result to show that certain singular value problems for tensors are NP hard (Theorem 17. 3|) . 

Problem 5.2 (TQF K ). Let A = £ M lxmxn be a real tensor, and let Ai(j, k) = a ijk , Bj(i, k) = 

aij k , and C k (i,j) = aij k be all the slices of A. Determine if the following set of equations 



(5.2) 



v T ^w = 0, 



has a solution u £ 



v £ 



l,...,l; 

l , w £ 



u Bjw 



0, j = 1, . . . ,m; u CfcV = 0, k 
with u ^ 0, v ^ and w^0. 



1, . . . ,n; 



When R is replaced by C, we call the corresponding problem tensor quadratic feasibility over C. 
Before approaching Problem 15.21 we consider the closely related triple quadratic feasibility. 



Problem 5.3 (3QF K ). Let Ai, Bi, Ci be real rax n matrices for each i = 1, . 
following set of equations 

(5.3) v T A;w = 0, i = l,...,n; u T Bj\v = 0, j = l,...,n; u T CfcV 
has a solution u, v, w £ M n with v ^ and ra^O. 



. , n. Determine if the 
0, k = 1, . . . ,n; 



Problem 15.31 is NP hard even on the subclass of problems where Bi = Ci for all i. It turns out 
that if one can show that triple quadratic feasibility is NP hard for Ai, Bi, and Ci coming from a 
tensor A with I = m = n, then checking if is the hyperdeterminant of a tensor is also NP hard. 
This is still an open problem. The proof of the following theorem is in the appendix. 

Theorem 5.4. Problem \5.3\ is NP hard. 

We close this section with a proof that tensor quadratic feasibility is a difficult problem. However, 
first we verify that the complexity of the problem does not change when passing to C. 

Lemma 5.5. Let A £ U lxmxn be a real tensor. There is a tensor B £ R 2 ' x2mx2ri suc h that tensor 
quadratic feasibility over M for B is the same as tensor quadratic feasibility over C for A 
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Proof. Consider the tensor B = [fry*-] E R 2/x2mx2n given by setting its slices Bi(j,k) = bijk as 
follows: 

'At 1 TO Ai 

-As ' l+i ~ Ai 



Bi 



,1. 



It is straightforward to check that real solutions to (j5.2[) for the tensor B correspond in a one- 
to-one manner with complex solutions to (|5.2p for the tensor .4,. □ 

Theorem 5.6. Graph 3-colorability is polynomial reducible to Problem 15.^1 Thus, Problem \5.2\ is 
NP hard. 

Proof. Given a graph G = (V, E) with |V| = A;, we shall form a tensor A = A{G) E Z /xmxn with 
/ = k(2k + 5) and m = n = (2k + 1) having the property that system (|5.2p has a nonzero complex 
solution if and only if G has a proper 3-coloring. An appeal to Lemma 15.51 completes the proof. 

Consider vectors v = (xi, . . . , Xk,yi, ■ ■ ■ , yk, t) T and w = (x\, . . . , x&, yi, • • • , Vk,i) T of indeter- 
minates. The 2x2 minors of the matrix formed by placing v and w side-by-side are k(2k + 1) 
quadratics. Thus, the set of complex numbers determined by the vanishing of all these polynomials 
can be described by equations v T Ai\v = 0, i = 1, . . . ,k(2k + 1), for matrices Ai E ^( 2fc + 1 ) x ( 2fc + 1 ) 
with entries in { — 1,0, 1}. By construction, if these equations are satisfied for some v,w/0, then 
there is a c E C \ {0} such that v = cw. 

Next, we write down the 3k equations v T ^4jW = for i = k(2k + 1) + 1, . . . , k(2k + 1) + 3k 
whose vanishing (along with the equations above) implies that the Xi are cubic roots of unity; see 
()3.ip . We also encode k equations v T yljW for i = k(2k + 4) + 1, . . . , k(2k + 4) + k whose vanishing 
says that x^ and Xj are different if E E (c.f. Remark 13. 8p . Finally, consider the tensor 

A(G) = la ijk j E Z /Xmx ™ given by a ijk = Mj, k). 

We verify that A has the claimed property. Suppose that there are three nonzero complex vectors 
u, v, w which satisfy tensor quadratic feasibility (over C). Then, from construction, v = cw and also 
v encodes a proper 3-coloring of the graph G. On the other hand, suppose that G is 3-colorable 
with a coloring represented using roots of unity (x\, . . . ,x k ) E C k (as discussed in Section [3]). 
Then, the vectors v = w = (x\, ■ ■ ■ ,x^,Xi , . . . ,x^ , 1) satisfy the first set of equations in (15. 2p . 
The other sets of equations define a linear system for the vector u consisting of 4k + 2 equations 
in / = k(2k + 5) > 4k + 2 unknowns. In particular, there is always a u solving them, proving that 
complex tensor quadratic feasibility is true for A. □ 



6. Tensor eigenvalue is NP hard 

The eigenvalues and eigenvectors of a symmetric matrix A E R nxn are the stationary values and 
stationary points of its Rayleigh quotient q(x) = x T Ax/x T x. Equivalently, one may look at the 
quadratic form x T ^4x = Y^j=i a ij x i x j constrained to the unit / 2 -sphere, 

(6.1) \\-x\\l=xl + xl + --- + xl = l. 

The stationary conditions of the Lagrangian L(x, A) = x T Ax — A(||x||2 — 1) at a stationary 
point (A,x) yields the familiar eigenvalue equation Ax = Ax which is then used to define eigen- 
value/eigenvector pairs for any square (not necessarily symmetric) matrices. 

The above discussion may be extended to yield a notion of eigenvalues and eigenvectors for a 
3-tensor A E ^ nxnxn , "We start from suitably constrained stationary values and stationary points 
of the cubic form ^4(x, x, x) = J27jk=i a ijk x i x j x k associated with a 3-tensor A E W nxnxn . What 
is not so certain is the appropriate generalization of the (|6.ip . One may retain the constraint (|6.ip . 
Alternatively, one may view (|6.ip as defining either the Z 2 -sphere or a unit sum-of-squares with a 
corresponding generalization to the / 3 -sphere, 

(6.2) ||x||| = |xi| 3 + |x 2 | 3 + • • • + |x n | 3 = 1, 
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or a unit sum-of-cubes, 

(6.3) x\ + x\ + ■ ■ ■ + xl = 1. 

Each of these choices has its advantage: condition (|6.2p defines a compact set while condition (|6.3p 
defines an algebraic set — both result in eigenvectors that are scale-invariant (i.e. if x is an eigen- 
vector, then so is cx for any c / 0); condition (|6.1|) defines a set that is both compact and algebraic 
but results in eigenvectors that are not scale-invariant. These were proposed independently and 
studied in [M1 I5B]. 

If we examine the stationary condition of the Lagrangian L(x, A) = ^4(x, x, x) — Ac(x) where 
c(x) is one of the conditions in f)6. 1[) . f)6. 1 [) . or (j6.3[) . we obtain the following definition. 

Definition 6.1. Let A £ 1 and x E R n \{0}. A is said to be an I 2 -eigenvalue and v an l 2 - 
eigenvector of A £ fl£ raxnxn if 

En 
. . a ijk XiXj = Xx k , k = l,...,n, 
1,3=1 

A is said to be an I 3 -eigenvalue and v an I 3 -eigenvector of A if 
(6.5) 2_^,, = ^a ijk x i x j = \x k , k = l,...,n. 

By (|2.3p . shorthands for f|6.4[) and (|6.5p are «4.(x, x,J n ) = Ax and A(x,x,I n ) = AZ(x, x, I n ). 

Problem 6.2 (Tensor eigenvalue). Given a 3-tensor A E M. nxnxn . Determine A E R such that 
(|6.4p or ()6.5p /ios a solution for some x E R n \{0}. 

By quadratic feasibility (Theorem 13. ip . we obtain the following result. 

Theorem 6.3. The tensor I 2 - and I 3 -eigenvalue problems are NP hard. 

Specifically, our methods show that determining if A = is an eigenvalue of a tensor is an NP 
hard problem. We remark that the situation is the same for determining whether any fixed rational 
number A is an / 3 -eigenvalue, which is easily seen from the form of equations (16. 5ft , 

7. Tensor singular value is NP hard 

The argument for deriving the singular value equations for a 3-tensor is similar to that for 
the eigenvalue equation in the previous section. It is easy to verify that the singular values and 
singular vectors of a matrix A E M. mxn are the stationary values and stationary points of the 
quotient x T J 4y/||x|| 2 ||y|| 2 . Indeed, at a stationary point (x, y, cr), the first order condition of the 
associated Lagrangian L(x,y, a) = x T Ay — o~(||x||2||y||2 — 1) yields the familiar singular value 
equations Av = cm, A T u = o~v where u = x/||x||2 and v = y/||y||2- 

This derivation has been extended to higher-order tensors to define a notion of singular values 
and singular vectors for tensors |34j . For A E ^ lxmxn j the Lagrangian L(x, y, z, a) = -4(x, y, z) — 
pII z IIp — 1) i s straightforward to define. The only ambiguity is in the choice of p. Let 
[n] := {1, . . . , n} for a positive integer n. 

Definition 7.1. Let a E R, u E R Z \{0}, v E R m \{0} ; w E R n \{0}. The number a is said to be 
an I 2 -singular value and u, v,w I 2 -singular vectors of A if 

l,m l,m l,m 

(7.1) ^2 a ijk UiVj = aw k , k E [n]; ^ a ijk VjW k = au k , i E [/]; ^ a ijk w k Ui = av k , j E [m]. 

i,j=l j,k=l i,k=l 

The number a is said to be an I 3 -singular value and x an I 3 -singular vector of A if 

l,m l,m l,m 

(7.2) a ijk UiVj = aw 2 k , k E [n]; a ijk VjW k = au 2 k , i E [/]; a ijk w k Ui = av 2 k , j E [m]. 



Note that the spectral norm ||«4||2 2,2 is either the maximum or minimum value of ^4(x, y, z) 
constrained to the set {(x, y,z) | ||x||2||y||2||z||2 = 1} and thus an Z 2 -singular value of A. 
We shall prove the following theorems in this section. 

Theorem 7.2. Determining whether a = 1 is an I 2 singular value of a tensor A is NP hard. 

Theorem 7.3. Determining whether a = is an I 2 or Z 3 singular value of a tensor A is NP hard. 

By homogeneity and the observation that o~ max (A) = ||-4.||2,2,2> the following is immediate. 

Corollary 7.4. Fix a G Q. Determining if a is an I 2 singular value of a tensor A is NP hard. 

We employ two very different methods to prove Theorems 17.21 and 17.31 For the first theorem, we 
reduce the decision problem to that of computing the max-clique number for a graph, extending 
some ideas from [37J and [25J. The proof of the second theorem, on the other hand, follows directly 
from Theorem 15.61 which was proved by a reduction to 3-colorability. 



Proof of Theorem 7.2. We cast (|4.2p in the form of a tensor optimization problem. Set An to be 
the three dimensional tensor with equal to the coefficient of the term UiVjWk in the multilinear 
form (|4.2|) . Then, Tt is just the maximum Z 2 -singular value of An- We show that if we could 
decide whether a = 1 is an I 2 singular value of Ae, then we could solve the max-clique problem 
(a well-known NP Complete problem). Given a graph G, construct the tensor At for each integer 
t G {1, . . . , n}. The largest value of £ for which 1 is a singular value of At is uj{G). To see this, 
notice that if t is larger than oj(G), the maximum singular value of At is smaller than 1 by Lemma 
14.41 Therefore, a = 1 cannot be a singular value of At in these cases. However, a = 1 is a singular 
value of the tensor A w . □ 

Using the techniques that prove Theorem 17.21 it is also possible to show that approximating the 
spectral norm of a tensor is NP hard. 

Corollary 7.5 (He-Li-Zhang |25j). Tensor spectral norm is NP hard. 

8. RANK-1 TENSOR APPROXIMATION IS NP HARD 
For r > 1, the best rank-r approximation problem for tensors does not have a solution in general 
min ||^4 — Aixi ® yi <g) zi — • • • — A r x r (g> y r ® z r \\F 

iYi 

because the set {A G ¥ lxmxn | rank(„4) < r} is in general not a closed set when r > 1. The 
following is a simple example based on an exercise in [30J. 

Examples 8.1. Let Xj, y, G F m , i = 1, 2, 3. Let A := x 1 <g> x 2 y 3 + xi <g> y 2 <8> x 3 + yi (g> x 2 <g> x 3 
and for n G N, let 

A n := xi <g> x 2 ® (y 3 - nx 3 ) + ^xi + ^Yi) ® ^ x 2 + ^Y2^ nx 3 . 

One may s/iow t/iai rank(A) = 3 iffxi,yi are linearly independent, i = 1,2,3. Since it is clear that 
rank(A n ) < 2 and lim n _> 00 A n = A, the rank-3 tensor A has no best rank-2 approximation. 

This phenomenon where a tensor fails to have a best rank-r approximation is much more wide- 
spread than one might imagine, occurring over a wide range of dimensions, orders, and ranks; 
happens regardless of the choice of norm (or even Bregman divergence) used. These counterex- 
amples occur with positive probability and in some cases with certainty (in p 2x2x2 ; no tensor of 
rank-3 has a best rank-2 approximation). We refer the reader to [16] for further details. 
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The set of rank-1 tensors (together with zero), {x ® y ® z | x G F , y G F m ,z G F ra }, is on the 
other hand closed. In fact the set is precisely the Segre variety in classical algebraic geometry. We 
would like to find the best rank-1 approximation 

min||^4 — x ® y <g> z\\p. 

By introducing an additional parameter a > 0, we may write the rank-1 term in a form where 
IHh = || y || 2 = INIb = 1. Then 

||.4 - ctx® y ®z\\ 2 F = \\A\\ 2 F - 2a(A,x<g>y<g)z) + a 2 . 

This is minimized when a = max|| x || 2= || y |i 2= || z .|| 2=1 (.A, x (g) y <8> z). But (A,x ® y ® z) = -4(x, y, z) 
and so o" = ||^4||2,2,2, which we already know is NP hard to compute. 

9. Tensor rank is NP hard over R and C 

Johan Hastad |24] has shown that any 3SAT boolean formula in n variables and m clauses has 
an associated (n + 2m + 2) x 3n x (3n + m) tensor and that the boolean formula is satisfiable iff 
the associated tensor has rank not more than 4n + 2m. A consequence is that tensor rank is NP 
hard over Q and NP complete over finite fields. 

Since the majority of recent applications of tensor methods are over R and C, a natural question 
is whether tensor rank is also NP hard over these fields for, say, tensors with rational entries. 
In other words, given a tensor A with rational entries and r G N, check if rankig(„4.) < r or if 
rankc(-4) < r (Hastad's result implies that checking rankQ(„4) < r is NP hard). 

Note that tensor rank depends on the base field; an example of a tensor with real entries whose 
rank over C is strictly less than its rank over R may be found in |16j . The same can happen for 
tensors with rational entries; the argument can be found in the appendix. 

Theorem 9.1. The rank over R of a tensor with rational entries can be strictly less than its rank 
over Q. 

We would like to provide here an addendum to Hastad's result by pointing out that his proof 
contains the following stronger result. In particular, tensor rank is also NP hard over R and C. 

Corollary 9.2. Tensor rank is NP hard over any field as long as one is able to compute field 
arithmetic in that field. 

Proof. Observe that the matrices Vi,Si,Mi,Ci in (23] that comprise the 'slices' of Hastad's tensor 
are defined with only the additive identity 0, the multiplicative identity 1, and the additive inverse 
of the multiplicative identity —1. Hastad's argument does not depend on any special properties 
of Q or finite fields but only on the field axioms and that 0^1 (his argument also works in 
characteristic 2 when —1 = 1). □ 

Appendix 

In this section, we include proofs of theorems that were omitted in the main body of text because 
of space considerations. 

Proof of Theorem \5.4\ By Theorem 13.21 it is enough to prove that a given quadratic feasibility 
problem can be polynomially reduced to this one. Therefore, suppose that A4 are given n x n 
real matrices for which we would like to determine if x T ^4jX = (i = 1, . . . ,n) has a solution 
R n . Let Eij denote the matrix with a 1 in the entry and 0's elsewhere. Consider a 
system S as in (|5.3p in which we also define 

B\ = E\\ and E>i = En — En for i = 2, . . . , n; 

C\ = E\\ and C, = En — En for i = 2, . . . , n. 
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We shall construct a decision tree based on the answer to feasibility questions involving systems 
having the form of S. This will give us an algorithm to determine whether the original quadratic 
problem is feasible. 

Consider changing system S by replacing B\ and C\ with matrices consisting of all zeroes, and 
call this system S' . We first make two claims about solutions to S and S' . 

Claim 1 : If S has a solution, then (ui, vi) = (0, 0). Moreover, in this case, w\ = 0. 

To show this, suppose first that S has a solution with u± = and v\ ^ 0. Then the form of 
the matrices Cj forces u 2 = • • • = u n = as well. But then u = 0, which contradicts S having a 
solution. A similar examination with u\ ^ and v\ = proves the claim. It is also easy to see 
that wi = in this case. 

Claim 2 : Suppose that S has no solution. If S' has a solution, then v = cu and w = du for some 

7^ c, d G R. Moreover, if 5" has no solution, then the original quadratic problem has no solution. 
To verify Claim 2, suppose first that S' has a solution u, v, and w but S does not. In that case 

we must have u% ^ 0. Also, v\ ^ since otherwise the third set of equations {u\Vi — UiV\ = 0}™ =2 
would force v = 0. But then v = cu for c = — and w = du for d = — as desired. On the other 
hand, suppose that both S and S' have no solution. We claim that x T ^4jX = (i = 1, . . . , n) has 
no solution x/0 either. Indeed, if it did, then setting u = v = w = x, we would get a solution to 
S' (as one can easily check), a contradiction. 

We are now prepared to give our method for solving quadratic feasibility using at most n + 2 
queries to the restricted version (Bi = d for all i) of Problem 15.31 

First check if S has a solution. If it does not, then ask if S' has a solution. If it does not, then 
output "infeasible". This answer is correct by Claim 2. If S has no solution but S' does, then 
there is a solution with v = cu and w = du, both c and d nonzero. But then x T AjX = for x = u 
and each i. Thus, we output "feasible". 

If instead, S has a solution, then the solution necessarily has Ui, itfi) = (0,0,0). Consider 
now the n — 1-dimensional system T in which Ai becomes the lower-right (n — 1) x (n — 1) block 
of Ai, and C% and Di are again of the same form as the previous ones. This is a smaller system 
with 1 less indeterminate. We now repeat the above examination inductively with starting system 
T replacing S. 

If we make it to the final stage of this process without outputting an answer, then the original 
system S has a solution with 

u\ = ■ ■ ■ = u n -\ = v\ = ■ ■ ■ = v n -\ = w\ = ■ ■ ■ = u> n -i = and u n , v n , w n are all nonzero. 

It follows that the (n, n) entry of each Ai (i = 1 . . . , n) is zero. Thus, it is clear that there is a 
solution x to the quadratic feasibility problem, and so we output "feasible" . 

We have therefore verified the algorithm terminates with the correct answer and it does so in 
polynomial time with an oracle that can solve Problem 15.31 □ 

Proof of Theorem \9.1\ We will construct such a tensor explicitly. Let x and y be linearly indepen- 
dent over Q. First observe that 

zi (g> z 2 (g> z 3 + zi (g> z 2 (g> z 3 = 2xi <8> x 2 ® X3 — 4yi (g> y 2 (g> x 3 + 4yi x 2 y 3 - 4xi y 2 y 3 

where z = x — \/2y for z = x + y/2y. For simplicity, we choose Xj = [1, 0] T , y« = [0, 1] T E Q 2 for 

1 = 1,2, 3. Let A be the tensor above. It is clear that rank]ft(^4) < 2. We claim that rankQ(j4) > 2. 
Suppose not and that there exist Uj = [aj,6j] T , v, = [ci,di] T G Q 2 , i = 1,2,3, with 



A = Ui u 2 u 3 + Vi v 2 v 3 . 

12 



This yields the following polynomial equations for unknown real quantities 01,02,03, b\,b 2 ,b 3 , 
ci,c 2 ,c 3 , and di,d 2 ,d 3 : 

010203 + C1C2C3 = 2, aib 2 a 3 + cid 2 c 3 = 0, fria 2 a 3 + dic 2 c 3 = 0, 

(9.1) hb 2 a 3 + did 2 c 3 = 4, 010263 + cic 2 d 3 = 0, ai6 2 ^3 + 61^2^3 = 4, 

b\a 2 b 3 + d\c 2 d 3 = 4, bib 2 b 3 + d\d 2 d 3 = 0. 

It is easily checked using Grobner basis methods that the ideal (over Q) generated by the polyno- 
mials defining these equations contains: 2c\ — d\, c\d 2 d 3 — 2. It follows that in order for there to be 
rational solutions, we must have ci = di = 0, a clear contradiction to the second equation. Thus, 
there cannot be any rational solutions to (|9.ip , □ 
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